Object image recognition and instant active response with enhanced application and utility

ABSTRACT

A device for detecting and marking differences between two substrates, such as images or written text. The device has an image capturing device that is in communication with image recognition software. Images of a secondary images are analyzed and compared to a primary image to detect differences between the two. Once a difference is detected, the device automatically disperses a marking material in the proximity of the detected difference.

FIELD OF THE INVENTION

The current invention relates to the field of automated review andresponse, more specifically to a novel method and apparatus forreviewing images to detect changes from a previous version andautomatically marking any detected changes.

BACKGROUND OF THE INVENTION

The decreased use of pesticides on the one hand and the decreasedeffectiveness of those which are in use has resulted in a disturbingproliferation of insects in food and in the home. Moreover, insects orother invading species frequently infest orchards or similar crops.

The invention described herein makes it possible to detect andimmediately dispose of thrips, mites and aphids, among other insects,including those invisible to the untrained and unpracticed eye.

Many such insects are very difficult to dislodge by washing and areresistant to pesticides. Large scale efforts such as by heating oftenharm the food or other material to be cleansed.

SUMMARY OF THE INVENTION

The invention disclosed herein is a computer image analysis system,which magnifies an image of a substrate to be checked for bugs or othersmall objects and is trained to recognize various bugs commonlyassociated with such substrates to be checked. If a bug is identifiedany of various action steps are taken in different embodiments of theinvention, including removing the bug by way of an action headassociated with the imaging device. In other embodiments of theinvention, the system is positioned on a movable platform to scan a widearea for offending objects and/or to perform mitigation actions once anoffending object is detected.

Still in other embodiments of the invention, the system is configured todetect differences between a secondary image and a reference images.That is, the system is configured to review images and compare thereviewed images to a previously captured image of the same area todetect differences between the two. In the event that a difference isdetected, an action head is automatically deployed to dispense markingmaterial in the image area or in the proximity of the image area wherethe difference was detected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic side view of a scanning device disposed above asubstrate to be checked according to an embodiment of the invention.

FIG. 2 is a flow chart showing software processing steps according to anembodiment of the invention.

FIG. 3 shows a side perspective view of a scanning device incorporatedwith a drone according to an embodiment of the invention.

FIG. 4 is a schematic diagram of an exemplary device control systemaccording to an embodiment of the invention.

FIG. 5 is a schematic diagram of a drone scanning system according to anembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will now be described withreference to the above-identified figures. However, the drawings and thedescription herein of the invention are not intended to limit the scopeof the invention. It will be understood that various modifications ofthe present description of the invention are possible without departingfrom the spirit of the invention. Also, features or steps describedherein may be omitted, additional steps or features may be included,and/or features or steps described herein may be combined in a mannerdifferent from the specific combinations recited herein withoutdeparting from the spirit of the invention, all as understood by thoseof skill in the art.

In one preferred embodiment, a lens is used to point at a leaf oflettuce and capture an enlarged image thereof via an image capturingdevice. The image may be stored in digital memory for later analysis orit may be analyzed in real time. In either case, the image is sent to aprocessor that is trained to recognize the general characteristics andcolor of the lettuce, and which is also trained to recognize physicalcharacteristics and features of insects typically found on lettuce. Theimage is magnified so that the presence of the insect, even if wellhidden, will be identified by comparison with a library of insectsstored in memory. In one embodiment, the processor does not positivelyidentify a bug or other identifiable foreign object, but it mayrecognize the object as foreign. For instance, a processor may containparameters of acceptable color values or hues for a specific substrateand if an object is outside of such parameters—software running on theprocessor determines the object as “foreign.”

The identification of the insect will immediately trigger a response inan action head which is attached to the lens housing and is capable ofmoving to the insect position, and removing or destroying the insectautomatically. In another embodiment of the invention, rather thanremoving an observed insect—an action head grips the piece of lettuceand discards it.

FIG. 1 shows a scanning device having a casing 12, which houses an imagerecognition system. A downward facing microscope, lens 14 or any suchimage capturing device and magnification device is located at a bottomportion of the scanning device. As shown, the lens 14 is directed at asubstrate 16, such as, for example, a piece of lettuce. The lensmagnifies a segment of a substrate to be checked and it feeds capturedimages to an image recognition system for image analysis. Images may bestored on a digital storage medium, among other storage systems ormedia.

It will be understood by those of ordinary skill in that the device 10may be provided with a plurality of differently powered lenses which maybe automatically adjusted when greater focusing ability is needed andany of different image capturing devices may be utilized, such as forexample, a camera or a video camera, or an array thereof. It will alsobe understood that the image recognition system need not be housedwithin the casing 12 of the device—but rather the image recognitionsoftware may be provided at a location that is distant from theimage-capturing device. In such embodiment, an image-capturing device(e.g. a microscope lens coupled to an image capturing system) isutilized to capture images. The images are then sent by a wired orwireless connection to an image classifier.

FIG. 1 shows an action head 18, which is provided at the distal end of amovable arm 20. The action head may be equipped with one or moreinstruments, such as a gripping device and/or a suctioning device. Inanother embodiment of the invention, the action head is provided with aheating element or such similar heat source—which can destroy a bug or asegment of lettuce when it is brought into direct contact therewith.

In one embodiment of the invention, the device 10 housing the lens 14and action head 18 is a handheld unit, which may be manually orautomatically moved across a stationary substrate such as a leaf oflettuce. In another embodiment the device 10 is mounted on a stationarysupport structure and a conveyor belt positioned below the devicedelivers items to be scanned below the microscope lens of the device.Still in other embodiments of the invention, the device is mounted to alinear motion track and it incrementally moves (for instance byincremental movements of a rack and pinion wheel controlled by acomputer) across a substrate to be searched. In one embodiment, thedevice 10 may be used for purposes of “surveillance.” In thisembodiment, the device is mounted in a fixed position. When an offendingobject (such as an insect) enters the field of vision of the lens and isrecognized as such by the image recognition system—a command is sent toactivate the action head 18 to eliminate and/or neutralize the offendingobject. It will be understood that as an alternative to eliminatingand/or neutralizing an offending object, the device could mark theoffending object for subsequent removal or remedial action. Inembodiments, device 10 may be a drone, which may be a remotelycontrolled and/or autonomously controlled vehicle (e.g., aircraft,ground vehicle). For example, an autonomous vehicle may operatedaccordingly to pre-programmed rules, such as navigation directions(e.g., coordinates or street directions), and/or logical rules to governoperation, such as obstacle avoidance rules and/or task execution rules(e.g., using a scanning or imaging device to assess various subjects),to name a few.

It will be further understood that the moveable arm described herein maybe its own detached unit, but which operates under the control of thesoftware, which software may be stored in memory on the device 10 andconfigured to run on one or more processors, or which software may beremotely located, such as on a remote server accessible via a datacommunication signals and/or data networks. An exemplary device controlsystem is described herein with respect to FIG. 4.

In an embodiment of the invention, action head 18 is mounted on exteriorsurface of a device such as a drone, a vehicle, or the like. In otherembodiments, action head 18 is attached to the distal end of a movablearm. It will be understood that a movable arm may be any of variousstructures such as, for example, one or more linear guide tracks, rackand pinion systems or such similar relative motion mechanism forsupporting and moving an action head is within the teaching of theinvention. The arm is movable in any of various directions by way ofball joints, linear motion tracks or other such similar movementsystems. When a bug or other offending object is detected by the imagerecognition system, the software is programmed to send a signal to themoveable arm. The moveable arm is then controlled by a softwareapplication and directed to the located bug. The action head is deployedto either destroy the bug as described above or to suction it off of thesubstrate. In one embodiment of the invention, rather than directing theaction head to a specific location—the moveable arm is directed to pushthe piece of lettuce (or other substrate) away, thereby discarding thesame or removing it from a batch.

The computer used to control operations, execute routines and store datamay comprise at least one or more processors and memory storage devices.The computer also may receive a number of inputs and outputs forcommunicating information externally.

It will be understood that the computer which operated the device mayoperate under the control of an operating system and softwareapplications, components and programs that execute the routines andsystems described herein. In general, the routines executed to implementthe embodiments of the invention, whether implemented as part of anoperating system or a specific application, component, program, object,module or sequence of instructions will be referred to herein as “thesystem”, or “software”. The software controls the image acquisition,image storage, image analysis and movements of the arm, action headand/or the movement of the device along a track.

It will be understood by those of ordinary skill in the art that theinventive device is an apparatus having image capturing capabilities andimage recognition capabilities coupled with software that is programmedto determine whether or not an object in an image field is an offendingobject. An “offending object” herein is any physical, identifiablestructure or shape that is targeted for action. Examples of offendingobjects may include, but are not limited to, insects, dirt, mold growth,plant features, product imperfections etc. The device is programmed totake an action once an offending object is detected. “Action” refers toany remedial steps taken by the device to eliminate or otherwise addressthe offending object. For example, in one embodiment of the invention,the action head 18 of device 10 advances to a location of an offendingobject and it records the spatial coordinates of the same. Thecoordinates are stored for later treatment and or elimination.

FIG. 2 shows a number of processing steps performed by the software inan exemplary embodiment of the invention.

The device is initiated and begins capturing images 22. The images aresent to an image recognition system which classifies various images 24.The classifier may be an algorithmic classifier or a neural networksystem. The image recognition system is trained to recognizemorphological/physical characteristics of bugs or other objects to bedetected. The image recognition may also be trained to detect pixelconcentrations which may indicate the presence of bug or other objectsof interest.

If an image is determined to be a bug 26, then the software performsfurther processing steps.

In one embodiment, the software sends a signal to the moveable arm 28,which directs the action head to the location of the bug to remove thesame according to the teachings described above.

In another embodiment of the invention, the software sends a signal tothe moveable arm to push aside the item 30 upon which the bug wasdetected.

Still in another embodiment of the invention, upon detecting a bug, thesoftware sends or sounds an alert to a human operator 32. The humanoperator may intervene to remove the bug or the item.

In another preferred embodiment, the device is designed to deal withonly one kind of insect on one kind of food or other material. A singlekind of action suitable for the situation will be built into thisdevice.

Extension is obvious to a more sophisticated device with software taughtto deal with many different kinds of foods and materials, to recognize arange of different insects or objects, and different means of removingthe insect, including vacuum, a glue head, an electrical charge,freezing, heat, or even a drop of powerful insecticide. Powerfulpesticides sprayed or deposited over a large area is harmful, but atargeted drop on the insect itself will dispatch the insect and notaffect the surrounding atmosphere.

In another embodiment, there will be an array of lenses and responseheads so that a sheet being inspected for bed bugs can be continuouslypassed under the array.

In another embodiment of the invention the inventive system may be usedto remove offending objects, such as bugs, from a fluid. In oneembodiment, an image capturing device is fixed above a channel offlowing liquid. The device may comprise an array of image capturingdevices or lenses suspended above a channel or similar fluid stream. Itwill be understood that in an embodiment of the invention, the actionhead may be a vacuum head or suction head such that when the imagerecognition system detects a presence of an offending object, thesoftware sends a command to the action head to vacuum an area of fluidin the vicinity of the offending object. The vacuum head or suction headwill draw in the offending object, and possibly, some of the surroundingfluid and discard the same.

The invention may be used in any of various environments in which subtlechanges need to be detected and then acted on. For example, thebeginning of a disease affecting trees or other plants and itssubsequent spread is often the result of an insect, beetle or bugpenetrating the bark or other surface and destroying the structure fromwithin. Detecting a presence of a specific kind of invader is virtuallyimpossible if it requires a human observer's continued closeobservation. The instant invention can be deployed in a manner whichdetects and acts whenever an invader is detected on the surface. Forexample, the software may be trained to detect specific bugs or locusts.Once detected, the software sends a command to spray an offendingsubstance or a pesticide.

In another embodiment of the invention, the software is programmed todetect swarms of bugs or flying objects—irrespective of the type of bugsor objects. In one example, the software is trained to detect aplurality of distinct moving objects within an area of interest. Once athreshold number of moving objects (e.g. >10) is detected, the softwarewill confirm a presence of a swarm and it will automatically sendinstructions to the action head to address the swarm. In one embodiment,the action head will spray a mist of water vapor or insecticide, smokelaced with insecticide, repellant or similar offending substances.Alternatively, the device will sound an alarm to disperse the swarm.

The invention can be modified to recognize the sign of incipient diseaseon the skin of a human being at a size almost invisible to the humaneye. It is evident that the principles of the invention can be readilyapplied to other areas where detection, recognition, and action of aflaw, intrusion, or incipient flaw at a stage where it is barelyvisible.

Depending on the specific use, it is obvious that the invention can beassociated with a variety of platforms, both mobile and stationary. Forexample, the image capturing lens and action head may be mounted to amovement mechanism such as a linear guide track, a pulley system, a rackand pinion or any such similar movement mechanisms. Alternatively, thedevice may be attached to or embedded within a drone, hovercraft,aircraft or similar dirigible. (Mechanical devices/mechanisms for movingthe device referred to as “movable platforms” herein.) In the embodimentwhereby the device is mounted to a movement mechanism, the software maybe programmed to move the device in any of various pre-determined orrandom movements. In an embodiment of the invention, once an offendingobject is detected, the software sends a command to interrupt movementof the device and deploy the action head to execute remedial actions.

Control of any of the above mobile or active platforms envisioned abovecan be implemented in a variety of ways, including voice recognition.

It is obvious that the insect image recognition and instant activeresponse device, as well as any attendant platform or support can besupplied with energy in a variety of ways, including batteries, solar,electromagnetic and hard wires, among others.

The instant invention is not limited to any specific materials ofconstruction or size, and is readily modified by change in programmedrecognition patterns to react to different insects, insect parts,plants, plant parts, and in one preferred embodiment, to detectextremely small predictable defects or imperfections, among others, inthe manufacturing process or in manufactured products.

In this latter use, it is obvious that more than one device can beconnected to work in tandem, or in any manner called for by thesituation.

In one preferred embodiment, a device in accordance with the inventionis furnished with assisted illumination to extend its use at night,through the use of light and infrared, among others. To extend its usefurther, x-ray and other surface penetrating radiation can be attachedto the platform or to the device itself.

It is obvious to someone skilled in the art that the image recognitionand instant response feature can be incorporated into or provided on theplatform.

In one preferred embodiment, a robot platform or movable platform may beprovided with an image magnification device to magnify an image of anarea, segment and/or substrate to detect objects it is trained todetect.

It will be understood that in addition to observing and capturingimages, the robot or movable platform may be programmed to provide anactive response to remove, mitigate and/or react to various conditions.Any of a variety of actions steps may be deployed by the robot such as,but not limited to, sending an alert or an update, and/or expelling aspray or substance such as pesticide, vapor or smoke. For example, inembodiments of the invention a movable platform such as a robot,vehicle, or drone is utilized to travel about a field, orchard or forestand obtain images of plants and/or trees growing therein. The softwaredetects any of various conditions associated with plants and/or treesand is trained to react accordingly. In one embodiment, the device istrained to detect boring insects (e.g. Emerald Ash Borers) byrecognizing physical features such as appearance, color, size, shapeetc. Additionally or alternatively, the device detects holes in plantsor trees created by such insects. The device then automatically respondsby directing the action arm to the detected insect or its entry hole.For example, in one embodiment, the device directs the action arm to anentry hole formed by boring insects and releases a blast of white paintor similar marking material to mark the tree for removal or for remedialtreatment.

In a further implementation of the invention, the device is programmedto provide an instant response which results in recognition and captureinstead of recognition and reject. That is, objects, such as insects,which are of interest instead of being offending, can be trapped andcaptured with the same device, using only an obvious modification. Thatis, rather than dispersing or deflecting an object of interest, thesoftware sends a signal to the mechanical arm to capture and maintainthe object of interest.

The inventive device may be deployed for use in quality controlactivities. In this embodiment, the device may be trained to recognizequalitatively acceptable objects and those that do not meet acceptablecriteria (or “defective objects”). The software is programmed to employan action head to capture defective objects. Acceptable criteria may beany of size criteria, shape criteria or such similar metrics calculatedby the software or algorithmic classifier. In other embodiments,acceptable criteria may be based on color criteria, pixel counts, pixelsaturation or any such similar image criteria the software and imagerecognition/analysis software is programmed to analyze.

It is obvious to anyone skilled in the art that the invention can bemodified to be used as a stand-alone, hand held device, or fixed inplace with items to be inspected passing through. It is also evidentthat lens/image capture device and action heads can be in a circular orother convenient pattern, and on both sides of the material, as on bothsides of a sheet.

In one preferred embodiment, the invention is attached to a drone whichis programmed to move up and down, and all around a tree periodically,subsequently to move on to other trees. This will enable the instantinvention to protect forests, orchards and plantations against invasivespecies. Drones can be programmed to travel in certain sectors of aforest or orchard, around a periphery or in any pattern as designated byan operator to capture images in the area below it and instantly react.

A drone can also be programmed to remain stationary, detecting andprotecting against the arrival of an expected invasive species for whichit has been trained. Similarly, attaching the device to a robot enablesthe protection against land based invasive species.

In each case, the proliferation of robots and drones, as well as otherplatforms, extends the use and effectiveness of the device. Included insuch other platforms are hovercrafts, extendable legs and floatabledevices among others known to those skilled in the art.

FIG. 3 shows a drone 34 used to capture images and provide an activeresponse according to an embodiment of the invention. Drone 34 has animage capturing device for capturing images of areas to be analyzed. Forexample, as shown, drone 34 has a lens 36 which is part of a camera orvideo camera. In a preferred embodiment, the image capturing device ishoused in a movable and/or rotatable housing. The lens 36 capturesimages, which are then analyzed by the image analysis software. Theimage analysis software may be located in a computer residing in thedrone 34 or images may be sent via wire or wireless communication to acomputer at another location.

An action arm 38 is shown extending from the body of the drone 34.Action arm 38 has one or more rotatable joints 40 a, 40 b, ball jointsor similar pivoting members allowing for various movement of the actionarm 38. For example, in the embodiment shown, action arm 38 is shownpointing downward in an orientation substantially orthogonal to the body41 of the drone, but it could be rotated around joint 40 a to a 90°angle.

An action head 42 is shown positioned at the terminal end of action arm38. The software is configured to direct action head 42 in the directionof a detected offending object and to automatically initiate remedialactions. For example, action head 42 is activated to expel any ofvarious substances described above in response to a command from thesoftware. A tank or similar storage reservoir within the drone storessubstances to be dispersed or dispensed from the drone.

In another embodiment of the invention, drone 34 is used to captureimages of plants or features of plants and to disperse pollen inresponse to such detection. In embodiments of the invention, the systemdetects images of plants such as flowers and trees to determine whetheror not the plant is a flowering plant adapted for receiving pollen.Additionally or alternatively, the system detects plant object orfeatures that are adapted to receiving pollen. Upon detection of suchplants and/or upon the detection of reproductive features of floweringplants—the drone automatically dispenses pollen. In embodiments of theinvention, the pollen is directed to the approximate location ofdetected flowering plants, but in other embodiments, the pollen isdirected to an area proximate to a detected flower or reproductivefeature.

In embodiments of the invention, an image capturing mechanism is used tocapture images of plants, trees or other vegetation and image analysissoftware is utilized to detect objects consistent with flowering plants.As will be understood by those of ordinary skill in the art, the imageanalysis software may be located in a computer residing in the drone 34or images may be sent via wire or wireless communication to a computerat a remote location. The image analysis software determines whether ornot a plant is one that is adapted to receive pollen and/or whether ornot a plant feature is an organ that is adapted to receive pollen (suchas a pistil).

In embodiments of the invention, the drone 34 may be directly controlledby a human operator, whereas, in other embodiments the drone iscontrolled by one or more computers. The drone 34 flies over areas ofvegetation and its image capturing system scans the terrain below. Inembodiments of the invention, the image analysis system is trained todetect specific plants for purposes of pollination. In otherembodiments, the image analysis system is additionally or alternativelytrained to detect specific plant features that are adapted for receivingpollen.

In embodiments of the invention, once a particular plant-type isdetected, the system is programmed to release pollen in the vicinity ofsuch detected plants. That is, once a plant of interest is detected, thesoftware sends a command to the drone to navigate toward such plants andto release pollen.

In other embodiments of the invention, the system is programmed todetect specific plant features, like pistils. Once a pistil is detected,the program sends a command to an action arm to release pollen in thedirection of the detected pistil. Action arm 38 is shown extending fromthe body of the drone 34. Action arm 38 has one or more rotatable joints40 a, 40 b, ball joints or similar pivoting members allowing for variousmovement of the action arm 38. For example, in the embodiment shown,action arm 38 is shown pointing downward in an orientation substantiallyorthogonal to the body of the drone, but it could be incrementallyrotated around joint 40 a to a 90° angle in order to more accuratelypoint the action head 42 in the direction of a pistil.

Action head 42, shown positioned at the terminal end of action arm 38,is provided with a nozzle or such similar spout for releasing a cloud,mist or similar stream of pollen. The software is configured to directaction head 42 in the direction of a flower to be pollinated andautomatically expel pollen in the direction of the plant of interest,flower of interest, or plant feature of interest. A tank or similarreservoir within the drone stores substances to be dispersed ordispensed from the drone.

In one exemplary implementation of the invention, the system isprogrammed to detect features associated with almond trees. In thisembodiment, once the software confirms a presence of an almond tree, itwill automatically send instructions to navigate the drone 34 to an areaproximate to the almond tree and subsequently send instructions to theaction head 42 to release pollen. In one embodiment, the software istrained to identify flowers on a almond trees and to disperse pollen onor near respective flowers. In other embodiments, the system is trainedto detect respective reproductive features on flowers of the almond tree(such as pistils) and the action arm is instructed to direct the actionhead 42 toward the reproductive features. Once the action head isproperly oriented, a command is sent to expel a spray or mist of pollen.

In a preferred embodiment of the invention, the software is trained torecognize a specific pistil and initiate a dispersal of pollen that isspecific to the pistil of interest. In another embodiment of theinvention, reservoirs of different pollen types are provided on thedrone or similar movable device. The software is trained to recognizeand detect a variety of different flowers/pistils (associated withdifferent flowers or flower types) and disperse a pollen type thatcorresponds to the detected pistil.

It will be understood by those of ordinary skill in the art that drone34 or a similar aircraft, hovercraft or dirigible having an imagecapturing device in communication with an image recognition system maybe used to detect and monitor any of various conditions and instantlyreact by dispersing any of various substances or performing otheractions via an associated action head. For example, a device may beprogrammed to detect plant conditions or soil conditions (e.g. usingcolor properties thereof) and to automatically disperse water ornutrients to the detected areas when a dry soil condition or anunhealthy plant condition is detected. In other embodiments, a devicemay be programmed to detect fires. For example, a drone may beprogrammed to fly over a forested area and detect visual indicia ofsmoke or fire. Additionally or alternatively, the device may have a heatsensor to detect fires. Once a fire is detected, the device isprogrammed to navigate into proximity of the fire and automaticallydisperse fire retardants such as chemicals or water.

It will be understood that the computer which operated the device mayoperate under the control of an operating system and softwareapplications, components and programs that execute the routines andsystems described herein. In general, the routines executed to implementthe embodiments of the invention, whether implemented as part of anoperating system or a specific application, component, program, object,module or sequence of instructions will be referred to herein as “thesystem”, or “software”. The software controls the image acquisition,image storage, image analysis and movements of the arm, action headand/or the movement of the device along a track or other movementmechanism.

FIG. 4 is a schematic diagram of an exemplary device control system 400.The device control system may comprise a computer system comprising oneor more computers. The device control system may govern operation of animaging and/or image evaluation device, as may be employed by an imagingdrone as described herein. In embodiments, certain components of thedevice control system 400 may be located on-board the device, such as onor within a drone, or remotely, such as at a remote computer system,which may be accessible via a data network. For example, the imageidentification module 436 and/or action module 438 may be locatedremotely, e.g., on one or more servers. Image data may be uploaded (viaphysical connection of memory storage devices and/or wirelessly) to theimage identification module 436 for evaluation. In embodiments, actioninstructions may be transmitted by an action module 438 to one or moredrones for execution.

The device control system 400 may comprise hardware, such as one or moreprocessors 402, a communication portal 404, one or more camera inputdevices 406, one or more sensor input devices 408 (e.g., scanners, rangefinders, position sensors (e.g., GPS receivers, altitude sensors, toname a few), action device 410 (e.g., action head and/or movable arm, asdescribed herein), and/or flight controls 412. Flight controls 412 caninclude thrusters, engines, motors, turbines, fans, rotors, propellers,thrust vectoring control surfaces, aerodynamic control surfaces, and/oractuators and/or servo motors to move such hardware components. Inembodiments, a drone can include wheels, treads and tracks, or otherground propulsion systems, including motors. In embodiments, the dronecan be designed to float and thus may include floatation devices (e.g.,pontoons) or buoyant exterior components of the drone, as well as waterpropulsion systems.

The device control system 400 may further comprise non-transitorycomputer-readable memory (e.g., local and/or remote), which may storeand/or access data, e.g., in one or more databases. Such data caninclude plant reference data 420, insect reference data 422, actionrules data 424, captured image data 426 or other sensor data, and/orposition data 428, as described herein. The device control system 400may also include one or more software module stored in the memory andconfigured to execute machine-readable instructions to perform one ormore processes. Such modules can include a flight control module 432,image capture module 434, image identification module 436, and/or actionmodule 438. The processes and functions described with respect to eachmodule may be performed by one or more other modules, such as othermodules described herein or additional modules.

The communications portal 404 may handle, process, support, and/orperform wired and/or wireless communications, such as transmittingand/or receiving data (e.g., data packets). In embodiments, transmissiondescribed with respect to a single data packet may comprise a pluralityof data packets. Data packets may be discrete electronic units of data.In other embodiments, transmissions may comprise non-discrete signals,such as data streams. Transmissions described with respect to datapackets may also comprise data transmissions via other communicationsmechanisms known in the art, such as data streams. Communications portal404 can comprise hardware (e.g., hardware for wired and/or wirelessconnections, such as communications chipsets, communications interfaces,and/or communications antennas, to name a few) and/or software.

Wired connections may be adapted for use with cable, plain old telephoneservice (POTS) (telephone), fiber (such as Hybrid Fiber Coaxial), xDSL,to name a few, and wired connections may use coaxial cable, fiber,copper wire (such as twisted pair copper wire), and/or combinationsthereof, to name a few. Wired connections may be provided throughtelephone ports, Ethernet ports, USB ports, and/or other data ports,such as Apple 30-pin connector ports or Apple Lightning connector ports,to name a few.

Wireless connections may include cellular or cellular data connectionsand protocols (e.g., digital cellular, PCS, CDPD, GPRS, EDGE, CDMA2000,1×RTT, Ev-DO, HSPA, UMTS, 3G, 4G, 5G, and/or LTE, to name a few),Bluetooth, Bluetooth Low Energy, Wi-Fi, radio, satellite, infraredconnections, ZigBee communication protocols, to name a few.Communications interface hardware and/or software, which may be used tocommunicate over wired and/or wireless connections, may compriseEthernet interfaces (e.g., supporting a TCP/IP stack), X.25 interfaces,T1 interfaces, and/or antennas, to name a few.

Turning to the data that the device control system 400 may store and/oraccess, plant reference data 420 can include one or more images of eachof a plurality of species for image comparison purposes and/or anidentifier or database association to indicate the respective speciesassociated with each image. In embodiments, the plant reference data caninclude images of plant parts, such as a pistil, petal, or leaf, to namea few. Plant reference data can also include growing condition data,which may be coupled with GPS data of captured images to narrow thenumber of reference images that are likely to produce a match. Growingcondition data can include any of water availability, soil type,temperature information (e.g., temperature ranges), climate, geographiclocation information, to name a few.

Insect reference data 422 can include one or more images of insects ofvarious species or insect components (e.g., wings) and an indicator orreference to associate each image with its respective species. Insectreference data may include size information (e.g., cross-sectionalmeasurements, measurements of body components, such as body segments,antennas, legs), body information (e.g., number of body segments, numberof antenna), color information, geographic information (e.g., indicatingwhere the insect is likely to be found), habitat information (e.g.,indicating habitats in which the insect is likely to be found, such asthey type of crops, type of terrain, temperatures), and/or food sourceinformation.

Action rules data 424 can comprise rules to control an action device 410(e.g., to control movement and/or usage of the action device 410) and/orlogical rules to govern when to use the action device 410.

Captured image data 426 can comprise one or more images (e.g., imagefiles), sequences of images, and/or videos (e.g., video files). Capturedimage data 426 may be associated with position data indicating aposition of the subject of the image and/or a position of the drone orcamera. The device control system 400 may further store and/or accessadditional sensor data from other sensor input devices 408, such asrange information (e.g., from the drone or camera to an image subject),infrared imaging data, heat imaging data, temperature information,and/or ambient light intensity information, to name a few.

Position data 428 can include global positioning coordinates (e.g.,indicating latitude, longitude, and/or altitude or elevation), streetaddress information, and/or local coordinate information (e.g., one, twoor three-dimensional locations in relation to the drone or camera).

A flight control module 432 may control movement of a drone, such as bycontrolling thrust, control surfaces or other flight control hardware412.

An image capture module 434 may govern when and how to capture images(e.g., which subject to focus on, zoom level, type of imagery to capture(still versus video), and/or number of images to capture, to name a few.

An image identification module 436 or image classifier may perform imageanalysis, such as comparisons to reference images and/or reference dataas described herein, to detect one or more subjects in a captured image,such as plant species, insect species, insect quantities, and/or otherforeign objects.

An action module 438 may evaluate action logical rules with respect tocaptured and processed image data to determine one or more actions totake. The action module may also control one or more action devices(e.g., such as an action head and/or movable arm attached thereto) tocause them to carry out the determined actions.

FIG. 5 shows a schematic diagram of an exemplary drone scanning system.The system can comprise one or more drones 502 (e.g., drones 502-1,502-2, . . . 502-N) and/or a control computer system 510, which may beremotely located, such as on one or more servers. The devices (e.g.,drones) and/or computer systems may be operatively connected directly,e.g., via wired or wireless communications, and/or indirectly, e.g., viaa data network 501, such as the Internet, a telephone network, a mobilebroadband network (e.g., a cellular data network), a mesh network, alocal area network (LAN) (including a wireless local area network, e.g.,a Wi-Fi network), a wide area network (WAN), a metropolitan area network(MAN), and/or a global area network (GAN), to name a few. Data networksmay be provided via wired and/or wireless connections. Data networks maybe public or private. Accordingly, data networks may be open or closed,such as requiring authorized access, specific communication connections,or specialized hardware and/or software. In embodiments, any combinationof communications channels may be utilized.

Processing of data from one or more drones 502 and/or control of eachdrone may be performed by one or more respective processors containedwithin or on each drone, one or more processors contained within or onone or more master drones that transmit commands to subordinate drones,and/or performed remotely such as at a remotely located control computersystem, which may be one or more servers comprising one or morecomputers that receive data from and/or transmit instructions to thedrones. In embodiments, any of the data processing and/or device controlfunctions may be divided among entities, such as the drones 502 andremote control computer system 510. For example, flight controls orvehicle movement may be handled at each device, while image processingmay be performed remotely. In embodiments, data acquisition may behandled at the device (e.g., capture of images and/or sensor data) andtransmitted to the remotely located control computer system 510. Thecomputer system 510 may process such data as described herein (e.g.,perform image recognition and/or determine actions), and/or transmitinstructions (e.g., action instructions, which may be machine-readableinstructions to execute one or more determined actions) to the device502 or to one or more other devices. Accordingly, one or more firstdrones 502 may acquire data while one or more second drones 502 mayexecute actions based upon determinations from the acquired data.

Both drones 502 and the control computer system 510 may include one ormore processors, memory devices storing data in non-transitorycomputer-readable memory, which data may be organized in one or moredatabases, and communication portals (e.g., communications antennasand/or chipsets, as described herein). Drones and the control computersystem may further comprise one or more input devices, e.g., to receivedirect user input. Accordingly, drones may have keypads, touch screens,buttons with hardwired or programmed functionality, microphones, cameras(e.g., with gesture processing software), or other input devices. Thecontrol computer system 510 may include one or more input devices suchas keyboards, mice, touchpads, touchscreens, microphones, cameras, toname a few, and/or output devices (e.g., display screens or speakers, toname a few).

Each drone 502 may also include the necessary respective hardware 504(e.g., cameras, sensors, vehicle propulsion and control hardware), data506 (e.g., rules for autonomous movement or control, flight path data,reference imagery and/or data, and/or captured data), and softwaremodules 508 (e.g., any of the software modules described with respect toFIG. 4) to operate in such a divided control system.

Similarly, the control computer system 510 will include the necessaryhardware 512 (e.g., processors, memory devices, and/or communicationportals), data 514, and/or software modules 516 running on one or moreprocessors to perform its assigned functions. Accordingly, remote data514 can include databases of reference imagery and/or other referencedata, which may be used in image analysis. Remote data 514 can alsoinclude a repository of captured images and/or other sensor data, suchas from across time periods and/or from a plurality of drones. Remoteprocessing modules 516 may include flight control modules, e.g., forcontrolling navigation routes or destinations, image analysis modules,and/or action modules to determine actions to take. The remote controlcomputer system 510 may receive data from one or more drones, store suchdata, process such data, and/or generate and transmit machine-readableinstructions to the drones.

It will be understood by those of ordinary skill in the art that oneinventive device is an apparatus having image capturing capabilities andimage recognition capabilities coupled with software that is programmedto determine whether or not an object in an image field is a pistil. A“pistil” herein is any physical, identifiable structure or shape of aplant part that is adapted to receive pollen. Once a pistil is detected,an automatic response in an action arm directs pollen to the identifiedpistil.

It will be understood that the device set forth herein may be programmedto provide an active response to remove, mitigate and/or react tovarious biological conditions. For example, in embodiments of theinvention, the device may be programmed to detect a presence of skinconditions and send an alert or expel a marking material to a body sitewhere a condition is identified. In an exemplary embodiment, the deviceis programmed to detect ticks embedded on user's skin by recognizingphysical features consistent with ticks, such as appearance, color,size, shape etc.

The tick-detecting device may be disposed on movable platform, forexample, as set forth herein, and programmed to move across an externalaspect of a person or of a limb. In other embodiments, the device is ahandheld unit that is grasped and manipulated by a user.

When a tick, bite, or mark is identified, the device automaticallyresponds by directing the action head to the location of the detectedtick and releases a blast of degradable ink, paint or similar markingmaterial to designate a need for careful inspection or removal as thecase may be.

In some embodiments of the invention, the device is programmable to betailored to identified users for more specific tick detection. In thisembodiment, the device is initially deployed to image the entire skinsurface of a given user. The device detects all images that contrastwith normal skin tone, and stores each of the images in a database (e.g.on a digital storage medium). Thus, after initial deployment, thesystem's database will have images of each mole, scar, or other darkmarks on the user's skin. Upon subsequent deployment, the system willsearch for contrasting images and compare each contrasting imagedetected against the stored database. If a match is found, then thesystem will determine that the detected image was previously present onthe user and no action is necessary. If, however, an image is detectedfor which there is no match—the system will then direct the action headto mark the newly found image. In this embodiment, a tick will presentas an image for which there is no corresponding match, and it willtrigger a response of the action head as set forth above.

Still in other embodiments of the invention, the device set forth hereinmay be programmed to provide an active response to remove, mitigateand/or react to various agricultural conditions.

For example, the device may be deployed in a field, garden, or orchardto detect early indications of weeds or similar harmful plants. Thedevice may be provided on a stationary structure to scan an area of afield. Alternatively, the device may be mounted on a movable platformsuch as a robot, vehicle, or drone that is utilized to travel about afield and obtain images of plants growing therein. The software isprogrammed to distinguish between a weed and a desired crop or plant.When it detects any of various features associated with weeds or otherundesired vegetation it will automatically react. In one embodiment, thedevice is trained to detect weeds by recognizing physical features suchas appearance, color, size, shape etc. In the event that weeds aredetected, the device automatically responds by directing the action armto the detected weed or plant and releasing herbicide to the vicinity ofthe detected weeds.

Still in other embodiments of the invention, the device is configured toanalyze images such as satellite or other aerial images of any ofvarious terrains and identify features of interest. Features of interestmay be any image segment that was not present in the same location in aprevious version of the image, or it may be an image or image segmentthat the device is trained to recognize.

For example, in one embodiment of the invention, the device isconfigured to detect differences between a first image and a secondimage taken from the same location. In this embodiment, a first image(e.g. digital photograph) is captured of a first area (e.g. ageographical location) at a first time and is stored in the systemmemory. A second image of the same type as the first image is capturedfor the same location at a later time. The device then analyzes thesecond image and compares each segment to corresponding segments on thefirst image. If a difference is detected—the action arm automaticallymoves to the area on the second image where a difference was detectedand expels a chalk or similar marking material, or it marks the areawith a marking device such as marker pen affixed to the end of theaction arm. It will be understood by those of ordinary skill in the artthat “differences” may include the addition of new image areas, theremoval of image areas or changes such as differences in color, texture,or the like.

The marking material may be stored as a liquid which may be sprayed by anozzle on the action head. Alternatively, marking material may be storedas fine solid particles or liquid droplets, in air or other gas andwhich may be deposited by way of aerosol spraying by the action head.

In embodiments of the invention, the device is configured to only markdifferences that are above a predetermined threshold value. For example,the device is configured to only react to changes that exceed a certainpixel count, pixel density, size, shape, color concentration, etc. asbetween a first image and an image acquired subsequently in time. Thatis, in embodiments of the invention, the tolerance for differences maybe set by allowing the user to adjust the threshold for difference thatwarrant marking by the action head.

For example, in an aerial photograph that contains a tree, a user maywant to note if the tree is removed. However, the user is uninterestedin the tree's natural changes, such as, natural growth, leave sheddingetc. In this regard, the device is configured to only mark changes thatexceed a threshold. In this embodiment, in the event that a tree isadded or removed, a sufficient number or concentration of pixels areinvolved so that the change amounts to a variation above the setthreshold. As such, the system will automatically mark the area in theaerial photograph where the tree was added or removed. In otherinstances, a lower threshold of variation is desired. For example, in asecurity surveillance environment, a user may want to be alerted to evenslight changes. In such environment, a user may set the tolerance todetect pixel changes associated with any object that is larger than abird or a rodent.

In exemplary embodiments of the invention, images of archeological areasor cemeteries under threat of development or encroachment will be takenby a satellite or aerial camera at a first time period and stored on thesystem's memory. At a subsequent time (e.g. a week), a second set ofimages is taken of the same geographic location and the second set ofimages is compared to the first set of images stored in memory. Changesto the corresponding images will cause an instant marking response. Thiswill enable the protection of sensitive and sacred areas known to beunder threat. Extension to monitoring of forests against illegallogging, or to the protection of game preserves against poaching isevident.

It will be understood that the image substrate upon which the device isemployed may be a physical printed image, such as a photograph or map,or it may be a digital image. It will be further understood that inembodiments of the invention any of various image types may be comparedsuch as radar images, X-ray images, CT scan images and the like.

It will be understood by those of ordinary skill in the art that thesystem may be used to detect and mark differences between any twosubstrates. In embodiments of the invention, the system storescoordinates for each of the primary image (i.e. taken earlier in time)and the secondary images (i.e. images acquired later in time than apreceding image), such that it may compare the secondary image to theprimary image at selected coordinates. Alternatively, the system istrained to match image features of one image with image features of asecond image without storing coordinates.

In other embodiments of the invention, the device is configured toautomatically detect differences in printed or written matter, such ashandwritten Torah scrolls. In embodiments of the invention, an image ofa reference scroll is stored in the device's memory. The device isconfigured to analyze a second scroll with respect to the referencescroll. In embodiments of the invention, the device is configured tomove in an x and y direction and image each letter in each row on eachpage or section of the scroll. For each letter, the device performs acomparison with a corresponding letter on the reference scroll stored inmemory. If a difference is detected, the action head is automaticallydirected to the area on the secondary image where the difference wasdetected and it expels a marking material in the area of the identifieddifference.

In embodiments of the invention, a comparison between letters of thereference scroll and second scroll include a checklist of imagecharacteristics. For example, when detecting an “aleph,” (first letterof Hebrew alpha bet), the device is configured to check whether or notthe corresponding aleph on the second scroll has an upper segment, alower segment, a diagonal segment and connecting segments. To the extentthat any of these image segments are missing, or unconnected the devicewill automatically mark the second scroll. The device is also configuredto detect areas within a letter that may be missing ink. For example,the device is configured to magnify each letter and determine itsoutside perimeter. Thereafter, the system scans areas within theperimeter that have a sharp contrast from dark to light. In embodiments,the system is configured to only detect areas of contrast that aresmaller than a predetermined threshold. This allows the system to markonly areas of missing ink or areas of cracked ink, but it will not marka letter having an open center. Such marking will, thus, notify a useras to at least any one of an insertion, deletion or misformation of aletter.

It should be understood that the preferred embodiment was described toprovide the best illustration of the principles of the invention and itspractical application to thereby enable one of ordinary skill in the artto utilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A device for detecting differences between twoimages, comprising: a digital memory device configured to store aprimary image; an image capturing device, the image capturing devicecomprising a lens and configured to capture a secondary image; an imagerecognition system analyze the secondary image to detect a differencebetween the primary image and the secondary image; a moveable arm, themoveable arm deployable to move to an area on the secondary image wherea difference was detected; an action head provided on the movable arm;and software configured to control the moveable arm, whereby thesoftware directs the action head to an area on the secondary image wherea difference is detected.
 2. The device of claim 1, further comprisingmarking material to mark an area on the secondary image where adifference was detected.
 3. The device of claim 2, whereby the actionhead provided on the moveable arm is configured to deposit markingmaterial on the secondary image at the area where a difference isdetected.
 4. The device of claim 2, whereby the marking materialcomprises an aerosol material that may be dispensed from the actionhead.
 5. The device of claim 2, whereby the marking material comprises amarking pen.
 6. The device of claim 1, whereby the primary image is anaerial photograph.
 7. The device of claim 1, whereby a differencecomprises any one of an addition of a new image area in the secondaryimage as compared to the primary image, removal of an image area in thesecondary image as compared to the primary image, and a change in colorin an area in the secondary image as compared to the primary image. 8.The device of claim 1, whereby the system is configured to detect adifference between primary image and the secondary image that exceeds apredetermined threshold value.
 9. The device of claim 1, whereby theprimary image is an image of a first written or printed text.
 10. Thedevice of claim 9, whereby the secondary image is an image of a secondwritten or printed text.
 11. The device of claim 9, whereby the systemdetects any one of a an insertion, deletion or misformation of a letterin the second written or printed text.
 12. The device of claim 11,whereby the action head is configured to automatically deposit markingmaterial on an area of the secondary image comprising any one of aninsertion, deletion or misformation of a letter in the second written orprinted text.